Papers by Katelyn X. Mei
Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation (2026.acl-long)
Copied to clipboard
Katelyn X. Mei, Yi-Li Hsu, Minjoon Choi, Zongwan Cao, Chenjun Xu, Bingbing Wen, Su Lin Blodgett, Lucy Lu Wang
| Challenge: | a large-scale analysis of human evaluation protocols for long-form generation tasks is lacking in current practice . current protocols lack proper standardization and operationalization, which can limit validity of evaluation . |
| Approach: | They conduct a large-scale analysis of human evaluation protocols for long-form generation tasks in *CL conference papers from 2023–2025. |
| Outcome: | The proposed evaluation protocols lack standardization and operationalization, the authors show . they also find that the evaluation protocols are inadequate for specific domains and tasks . |